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All claims being allowable, PROSECUTION ON THE MERITS IS (OR REMAINS) CLOSED in this application. If not included 
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NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative 
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EXAMINER'S AMENDMENT 



1 . An examiner's amendment to the record appears below. Should the changes and/or 
additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 
1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the 
payment of the issue fee. 

2. Authorization for this examiner's amendment was given in a telephone interview with 
Applicants Attorney (Thomas R.FitzGerald, Reg No. 26,730), on April 3, 2006, without traverse. 

3. The application has amended as follows: attach paper 



In the claims 
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REASONS FOR ALLOWANCE 



4. The following is an examiner's statement of reasons for allowance. 

Based on telephone interview and fax inquiry, filed 3/3//2006, see page 4 through page 
13, of the remarks, with respect to claims 11-14, 17-34, 37, 39-53 and 55-60 have been fully 
considered and are persuasive and upon further consideration the 103(a) art rejection of claims 
11-14, 17-34, 37, 39-53 and 55-60, has been withdrawn. 

5. Claims 1 1-14, 17-34, 37, 39-53 and 55-60 now renumbered as 1-44 is allowed. 

Based on applicant's amendment, claim 1 1 representing claim 51, the closest prior art of 
record (Maurer and Singh) references do not disclose or suggest, among other things, " marking 
salient features of an object with markers for motion tracking, acquiring a plurality of initial 2-D 
images of the object and calculating the 3D locations of the local markers in each action state by 
estimating the orientation and position of the face in each 2-D image of the action state to 
conform to the 3-D and 2-D location of the global markers under a perspective projection model 
and calculating the 3-D locations of the local markers to conform to the estimated orientation and 
position of the face and 2-D location of the markers under a perspective projection model. 

These key features in combination with the other features of the claimed invention are 
neither taught nor suggested by (Maurer and Singh) prior art of record. 

Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the issue 
fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 
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Contact Information 



6. 



Any inquiry concerning this communication or earlier communications from the 



examiner should be directed to Seyed Azarian whose telephone number is (571) 272-7443. The 
examiner can normally be reached on Monday through Thursday from 6:00 a.m. to 7:30 p.m. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu, can be reached at (571) 272-7429. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application information Retrieval (PAIR) system. Status information for published application 
may be obtained from either Private PAIR or Public PAIR. 

Status information about the PAIR system, see http:// pair-direct.uspto.gov. Should you have 
questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 
866-217-9197 (toll-free). 



Seyed Azarian 
Patent Examiner 
Group Art Unit 2624 
March 27, 2006 
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IN THE CLAIMS 

Please amend the claims as indicated below. 

Claims 1-10. (Cancelled) 

1 1. (previously presented) A method for tracking motion of a face comprising the steps— 
of: 

determining the calibration parameter of a camera; 
marking salient features of an object with markers for motion tracking; 
acquiring a plurality of initial 2-D images of the object; 
calculating 3-D locations of the salient features of the object in accordance with 
the calibration parameter of the camera; 

calculating 3-D locations of the global and local markers in the neutral state; and 
calculating 3-D locations of the local markers in each action state; 
receiving a chronologically ordered sequence of 2-D images of the object; 
storing or transmitting tracked motion of the object. 

calculating the 3-D locations of the local markers in each action state by 
estimating the orientation and position of the face in each 2-D image of the action state to 
conform to the 3-D and 2-D locations of the global markers under a perspective 
projection model and calculating the 3-D locations of the local markers to conform to the 
estimated orientation and position of the face and the 2-D locations of the local markers 
under a perspective projection model. 

12. (previously presented) The method of claim 1 1 further comprising the steps of: 

locking onto the markers; and 

detecting loss of lock and hence the need for re-locking onto the markers. 

13. (previously presented) The method of claim 1 1 wherein the step of tracking 
comprises the steps of. 



determining a surface normal for each salient features; 
tracking motion of the markers on the object in each 2-D image; and 
tracking the 3-D global motion of the markers on the object in each image; and 
tracking the 3-D local motion of the markers on the object in each image. 

14. (previously presented) The method of claim 12 comprising the further step of 
repeating the locking and tracking steps after the detecting step. 

15. Cancelled. 

16. Cancelled. 

17. (previously presented) The method of claim 1 1 wherein a first set of markers 
identifies global motion and a second set of markers identifies local motion of the 
face. 

18. (original) The method of claim 17 wherein the markers comprise at least two 
colors. 

19. (original) The method of claim 18 wherein the two colors are contrasting. 

20. (original) The method of claim 19 wherein the colors are black and white. 

21. (amended) The method of claim 4$ U wherein the markers comprise two 
concentric circles of different colors. 

22. (original) The method of claim 21 wherein the outer circle has a diameter at least 
twice the diameter of the inner circle. 



23. (original) The method of claim 1 1 wherein the step of selecting comprises 
wearing a head-set with markers. 



24. (original) The method of claim 23 wherein the head-set comprises a strap for a 
chin. 

25. (original) The method of claim 23 wherein the head-set comprises a strap for 
eyebrows. 

26. (original) The method of claim 23 wherein the head-set comprises at least one 
strap for a skull. 

27. (original) The method of claim 1 1 wherein the acquired 2-D images include at least 
two views of the face with markers in a neutral state at different orientations; 

28. (original) The method of claim 27 wherein the two views are orthogonal. 

29. (original) The method of claim 1 1 wherein the acquired 2-D images comprise front, 
forehead, chin, angled-right, angled-right-tilted-up, angled-right-tilted-down, angled-left, 
angled-left-tilted-up, angted-left-tilted-down, full-right-profUe, full-right-profiletilted-up, 
full-right-profile-tilted-down, full-left-profile, fulHeft-profile^tilted-up, and full-left- 
profile-tilted-down views of the face with markers in the neutral state, 

30. (original) The method of claim 1 1 wherein the acquired 2-D images comprise front, 
forehead, chin, full^right-profile, and full-left-profile views of the face with markers in 
the neutral state. 

3 1 . (previously presented) The method of claim 1 1 wherein the acquired 2-D images 
include a plurality of views of the face with markers in at least one action state. 

32. (original) The method of claim 31 wherein the action states of the face comprise 
smiling lips, kissing lips, yawning lips, raised eyebrows, and squeezed eyebrows. 



33 . (original) The method of claim 31 wherein the acquired 2-D images of the face in an 
action state include at least two views at different orientations. 

34. (original) The method of claim 33 wherein the two views are front and angled-right. 

35. Cancelled. 

36. Cancelled. 

37. (previously presented) The method of claim 1 1 wherein the step of calculating the 3- 
D locations of the global and local markers in the neutral state comprises the steps of: 

calculating the 3-D locations of the markers to conform to their 2-D locations in 
the 2-D images of the face in the neutral state under an orthographic projection model; 

calculating relative distances of the face to the camera in the 2-D images to 
conform to the 2-D locations of the markers and their calculated 3-D locations under a 
perspective projection model; 

modifying the 2-D locations of the markers to conform to the calculated relative 
distances and the 3-D locations under a perspective projection model; 

recalculating the 3-D locations of the markers to conform to their modified 2-D 
locations under an orthographic projection model; 

repeating the steps of calculating the relative distances, modifying the 2-D 
locations, and recalculating the 3-D locations to satisfy a convergence requirement; and 

translating and rotating the 3-D locations so that they correspond to a frontal- 
looking face. 

38. Cancelled. 

39. (original) The method of claim 13 wherein the step of tracking the 3-D global motion 
comprises the steps of: 

predicting the location of global salient features in a 2-D image; 
detecting global salient features in the 2-D image; and 



estimating the 3-D global motion of the face in the 2-D image. 

40. (original) The method of claim 39 wherein the step of predicting comprises 
calculating 2-D locations of the global salient features under a perspective projection 
model using the position and orientation of the face in a previous 2-D image, and the 
step of detecting comprise detecting the global markers. 

41. (original) The method of claim 40 wherein detecting the global markers comprises: 

determining visibility indices of global markers; 
designing correlation filters for the global markers; 
detecting the global markers by applying elliptical correlation filters in a 
neighborhood of the global markers; and 

eliminating superfluous and multiple detected locations. 

42. (previously presented) The method of claim 40 wherein the step of estimating the 3 
D global motion comprises calculating the position and orientation of the face to 
conform to the 3D locations and the detected locations of the global markers under a 
perspective projection model. 

43. (original) The method of claim 13 wherein the step of tracking the 3-D local motion 
comprises the steps of: 

predicting the location of local salient features; 
detecting local salient features; and 
estimating the 3-D local motion of the face. 

44. (original) The method of claim 43 wherein the local markers are placed on 
eyebrows and lips. 

45. (original) The method of claim 44 wherein the locations of the local markers 
comprise proximate ends of the eyebrows, comers of the hps, and the upper and lower 
centers of each lip. 



46. (previously presented) The method of claim 45 wherein (he-step of predicting the 
locations of local markers comprises calculating the locations of the local markers 
using the position, orientation, and action states of the face in a previous 2-D image and 
the step of detecting comprise detecting the local markers, 

47. (previously presented) The method of claim 46 wherein detecting the local markers 
comprise: 

determining visibility indices of local markers; 
designing correlation filters for the local markers; 
detecting the local markers by applying elliptical correlation filters in a 
neighborhood of the local markers; and 

eliminating superfluous and multiple detected locations. 

48. (previously presented) The method of claim 47 wherein the step of estimating 
comprises: 

finding 3-D locations of local markers to conform to the detected 2-D locations 
of the local markers; 

calculating an action vector representing the weights of the facial actions in the 
2-D image conforming to the found 3-D locations of the local markers and the 3-D 
locations of the local markers for the neutral and the action states under a perspective 
projection model. 

49. (original) The method of claim 48 wherein the step of calculating an action vector 
comprises the steps of: 

calculating the difference between the 2-D locations of the local markers detected 
in an image and the 2-D locations of the same markers corresponding to the neutral face; 

modifying the difference to conform to the orthographic projection; 

calculating the 3-D displacements of the local markers with respect to their 
location in the neutral face; and 

calculating the amount of facial actions conforming to the 3-D displacements of 
the local markers. 



-50. (original) The method of claim 4S wherein the step of calculating an action vector 
comprises the steps of: 

calculating the 2-D locations of the local markers corresponding to the neutral 
face using the global motion found for the current image; 

calculating the 2-D locations of the local markers corresponding to the action 
faces using the global motion found for the current image; 

calculating the distance between the detected locations, the distance between the 
neutral locations, and the distance between the action locations of the markers at the right 
and left corners of the lips; 

calculating the distance between the detected locations, the distance between the 
neutral locations, and the distance between the action locations of the markers at the 
upper and lower center of lips; 

calculating the distance between the detected locations, the distance between the 
neutral locations, and the distance between the action locations of the markers at the 
proximate ends of the eyebrows; 

determining the fractional displacements of the local markers for the lips area and 
for the eyebrows area; and 

determining action mode and amount for the lips area and for the eyebrows area 
based on the fractional displacements of the local markers. 

51. (previously presented) A method for tracking motion of an object in a chronologically 
ordered sequence of 2-D images of the object comprising the steps of: 

selecting global and local salient features of the object for tracking by fixing 
markers to the object; 

calculating 3-D locations of the markers at the global and local salient features for 
a neutral state of the object, and calculating 3-D locations of the local salient features for 
action states of the object; 

calculating the 3-D locations of the global and local markers in a neutral state; and 

calculating the 3-D locations of the local markers in each action state by 
estimating the orientation and position of the face in each 2-D image of the action state to 
conform to the 3-D and 2-D locations of the global markers under a perspective 



"pmjeNCtibn model and calculating the 3-D locationslaf llie local markers to conforin to the 
estimated orientation and position of the face and the 2-D locations of the local markers 
under a perspective projection model; 

predicting 2-D locations of the markers at the global and local salient features in a 
2-D image; 

detecting 2-D locations of the markers at the global and local salient features in 
the 2-D image; and 

estimating the global and local motion of the object in the 2-D image. 

52. (amended) The method of claim 6© 51 where the action states of the object define 
maximum local motions of the object. 

53. (amended) The method of claim #1- 52 where the method of calculating comprises: 

acquiring a plurality of initial 2-D images of the object in neutral state; 

acquiring a plurality of initial 2-D images of the object in action states identifying 
the 2-D locations of the salient features in each initial 2-D image; 

estimating the orientation and position of the object in each initial 2-D image to 
conform to the 2-D locations of the salient features under a perspective projection model; 
and 

calculating 3-D locations of the salient features to conform to the 2-D locations of 
the salient features under a perspective projection model, 

54. Cancelled. 

55. (amended) The method of claim 63- 51 wherein a first set of markers define global 
salient features and a second set of markers define local salient features. 

56. (amended) The method of claim 6* 51 wherein the step of estimating global motion 
comprises calculating 3-D position and orientation of the object to conform to the 3-D 



Jocations of the global markers in tta neutral state and thedetected 2-D locations of the 
global markers under a perspective projection model. 

57. (amended) The method of claim 63- 51 wherein the step of estimating local motion 
comprise calculating a vector of weights representing fractions of maximum actions of 
the object conforming to the 3-D locations of local markers in the neutral and action 
states and the detected 2-D locations of the local markers under a perspective projection 
model. 

58. (amended) The method of claim 60 51 wherein the object is a face. 

59. (amended) The method of claim 63- 5X wherein the markers comprise concentric 
circles with two contrasting colors. 

60. (amended) The method of claim 6& 59 wherein the step of detecting comprises 
applying elliptical correlation filters in a neighborhood of the markers. 



